Initial efforts to introduce quantization to native Hugging Face models that are already supported in optimum-executorch. Start with decoder-only text models using "8da4w" for linear layers and int8 for embedding.

Experiment the quantization configs with the following models:

• Qwen3-0.6B
• gemma-3-1b
• HuggingFaceTB/SmolLM2-135M

Example usage

via optimum-cli
optimum-cli export executorch --model Qwen/Qwen3-0.6B --task text-generation --recipe xnnpack --use_custom_sdpa --qlinear --qembedding --output_dir qwen3_8da4w_8we

or use the ExecuTorchModelForCausalLM.from_pretrained.

et_model = ExecuTorchModelForCausalLM.from_pretrained("./qwen3_8da4w_8we")

.pte size comparison:

qwen3_8da4w_8we:
total 1035336
-rw-r--r--  506M  May  1 13:09 model.pte

qwen3_8da4w:
total 1944584
-rw-r--r--  950M  May  1 13:16 model.pte

qwen3_float16:
total 2937408
-rw-r--r--  1.4G  May  1 13:22 model.pte

qwen3_float32:
total 5873128
-rw-r--r--  2.8G  May  1 13:26 model.pte